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Evaluating  After-School  L;are 


More  and  more  children  with  mothers  who  work  out¬ 
side  of  the  home  are  participating  in  after-school  pro¬ 
grams,  and  increased  federal  and  state  funding  suggests 
that  the  number  of  such  programs  will  continue  to  grow  in 
the  coming  years.  Funding  for  the  U.S.  Department  of 
Education's  21st  Century  Community  Learning  Centers 
alone  increased  from  $1  million  in  1997  to  $450  million  in 
2000,  with  $1.5  billion  proposed  for  2002. 

The  impetus  for  this  growth  extends  beyond  increased 
government  funding.  For  instance,  a  partnership  between 
the  U.S.  Department  of  Education  and  the  Mott  Founda¬ 
tion  has  given  rise  to  the  Afterschool  Alliance,  a  coalition 
of  public  and  private  organizations  dedicated  to  raising 
awareness  of  the  importance  of  after-school  programs  and 
advocating  on  their  behalf.  The  alliance's  goal  is  that  every 
child  should  have  access  to  high-quality,  affordable  after¬ 
school  care  by  2010. 

The  "high-quality"  goal  is  particularly  daunting.  The 
need  for  after-school  services  became  apparent  so  quickly 
that  there  has  been  virtually  no  time  to  muster  support  for 
testing  and  evaluating  the  various  aspects  of  the  programs 
being  offered.  Program  managers  who  are  committed  to 
high-quality  care  have  thus  been  faced  with  the  challenge 
of  attempting  to  measure  their  activities  against  almost 
nonexistent  standards. 

It  was  within  this  context  that  a  team  of  RAND 
researchers  sought  to  establish  a  set  of  model  after-school 
program  features  or  practices  (see  the  box)  that  are  sup¬ 
ported  by  scientific  studies  where  possible,  or  by  expert 
judgment  where  not.  A  second  objective  was  to  provide 
the  rating  scales  and  data  collection  instruments  required 
for  evaluators  to  test  programs  against  model  practices. 

The  research  team  began  by  systematically  assessing 
the  published  research  literature  on  after-school  care  to 
identify  candidate  model  practices.  They  were  defined  as 
program  or  process  elements  shown  or  upheld  by  experts 
in  the  field  to  be  associated  with  high-quality  after-school 
programs  or  positive  child  outcomes  (such  as  those  associ¬ 


ated  with  educational  attainment,  emotional  development, 
and  health).  The  researchers  found  only  a  few  scientifi¬ 
cally  sound  empirical  studies  that  demonstrated  an  associ¬ 
ation  between  certain  practices  and  desirable  outcomes. 
Most  of  the  publications  in  the  literature  simply  summa¬ 
rized  recommendations  provided  by  expert  panels  and 
individuals.  Therefore,  the  RAND  list  of  model  practices 
must  be  considered  preliminary  and  subject  to  change  in 
light  of  new  research. 

Model  After-School  Practices  Proposed  by 
RAND  Researchers 

Staff  Management  Practices 

Hiring  and  retaining  educated  staff 
Providing  attractive  compensation 
Training  staff 

Program  Management  Practices 

Ensuring  that  programming  is  flexible 

Establishing  and  maintaining  a  favorable  emotional 
climate 

Establishing  clear  goals  and  evaluating  programs 
accordingly 

Having  a  mix  of  younger  and  older  children 
Keeping  total  enrollment  low 
Maintaining  a  low  child-to-staff  ratio 

Maintaining  continuity  and  complementarity  with 
regular  day  school 

Paying  adequate  attention  to  safety  and  health 
Providing  a  sufficient  variety  of  activities 
Providing  adequate  space 

Providing  age-appropriate  activities  and  materials 
Providing  enough  quality  materials 

Communications  with  Other  Organizations 

Involving  families 

Using  community-based  organizations  and  facilities 
Using  volunteers 

Note:  Italics  indicate  strong  support  in  the  research  literature. 


Because  most  of  the  literature  was  at  varying  levels  of 
reliability  below  the  "gold  standard"  of  experimental 
research,  the  RAND  team  placed  particular  emphasis  on 
finding  a  way  to  convey  the  degree  of  support  for  each 
candidate  practice.  To  rate  the  candidate  model  practices 
in  this  regard,  the  RAND  researchers  adapted  conven¬ 
tional  meta-analytic  techniques  for  application  to  an 
empirically  weak  field. 

In  conventional  meta-analysis,  the  effect  sizes  found 
in  different  studies  are  weighted  according  to  the  reliabil¬ 
ity  or  scientific  rigor  of  each  study  and  then  averaged.  In 
the  present  case,  RAND  also  classified  studies  by  reliabil¬ 
ity  but,  instead  of  using  effect  sizes,  the  values  to  be 
weighted  were  the  frequencies  with  which  effects  were 
reported  (or  judged  to  exist).  The  final  scores  were 
expressed  in  qualitative  terms — that  is,  whether  a  model 
practice  candidate  had  strong,  moderate,  or  limited  sup¬ 
port  in  the  literature. 

The  candidate  practices  were  classified  by  type  and 
rated  according  to  the  level  of  support  for  each.  Those 
shown  in  italics  in  the  box  were  judged  to  have  strong 
support.  All  others  listed  in  the  box  had  moderate  sup¬ 
port.  Two  other  candidates — hiring  and  retaining  experi¬ 
enced  staff  and  keeping  the  turnover  rate  low — were 
judged  as  having  limited  support  and  were  dropped  from 
the  list  of  model  practices. 

As  a  further  aid  to  evaluators,  the  RAND  research 
team  devised  scales  to  help  evaluators  determine  whether 
their  program's  adherence  to  each  model  practice  would 
best  be  described  as  "excellent,"  "good,"  "minimal,"  or 


"inadequate,"  These  scales  are  specific  to  each  practice — 
for  example,  for  the  staff  management  practice  providing 
attractive  compensation,  the  break  points  for  excellent,  good, 
and  minimal  are  the  75th,  50th,  and  25th  percentiles  of 
hourly  wages  paid  to  child-care  workers  in  the  program's 
county. 

The  researchers  cautioned  that  in  assessing  adherence 
to  model  practices,  program  evaluators  should  be  sensi¬ 
tive  to  trade-offs  and  constraints  arising  from  a  program's 
objectives  or  budget.  For  example,  if  a  provider  sees  its 
mission  as  offering  after-school  care  to  all  children  in  the 
community  who  need  it,  low  total  enrollment  might  not  be 
an  important  goal.  High  enrollment  would  therefore  be  a 
disadvantage  that  is  deliberately  tolerated  to  permit  what 
the  provider  regards  as  the  greater  value  of  broad  commu¬ 
nity  service.  For  other  programs,  the  principal  objective, 
for  example,  may  be  to  keep  children  safely  off  the  streets 
after  school.  In  such  cases,  model  practices  intended  to 
enhance  educational  achievement  may  not  be  affordable 
or  actively  pursued.  The  evaluator  should  nevertheless 
assess  the  program's  adherence  to  all  model  practices,  but 
the  trade-offs  and  constraints  that  program  managers  face 
should  still  be  identified. 

The  scales  used  to  determine  adherence  to  model 
practices,  the  survey  forms  for  collecting  data  supporting 
application  of  the  scales,  and  the  results  of  a  sample  appli¬ 
cation  can  be  found  in  the  RAND  report  cited  at  the  end  of 
this  brief.  This  study  was  funded  by  Stone  Soup  Child 
Care  Programs,  a  provider  of  after-school  care;  by  RAND, 
through  its  Child  Policy  Project;  and  by  the  Promising 
Practices  Network. 


RAND  research  briefs  summarize  research  that  has  been  more  fully  documented  elsewhere.  This  research  brief  describes  work  conducted  as  part  of 
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NBN.  RAND®  is  a  registered  trademark.  RAND  is  a  nonprofit  institution  that  helps  improve  policy  and  decisionmaking  through  research  and  anal¬ 
ysis;  its  publications  do  not  necessarily  reflect  the  opinions  or  policies  of  its  research  sponsors. 
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